Papers by Sarah B. Lawsky
OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning benchmarks are expensive to build and ill suited for isolating specific failure modes. |
| Approach: | They propose a framework and benchmark for diagnostic evaluation of legal reasoning that uses symbolic representations of U.S. Bankruptcy Code statutes to generate large space of reasoning tasks and their machine-computable solutions on demand. |
| Outcome: | The proposed framework and benchmark provides diagnostic insights into the competencies and failure modes of language models. |